System and method for supporting state synchronization in a network environment

ABSTRACT

A system and method can support network management in a network environment. The network environment can include a plurality of configuration daemons (CDs), wherein each CD resides on a switch in the network environment. The CD operates to receive a configuration file that includes a list of known management key (M_Key) values. Furthermore, the CD operates to store the configuration file, and make the configuration file available to a local subnet manager(SM) on the switch, wherein the local SM is associated with a currently used M_Key value. Then, the CD operates to update the local SM with a new M_Key, after receiving an instruction from a master CD that is associated with a master SM in the network environment.

CLAIM OF PRIORITY

This application claims priority on U.S. Provisional Patent ApplicationNo. 61/645,517, entitled “SYSTEM AND METHOD FOR PROVIDING SECRETMANAGEMENT KEY IN A MIDDLEWARE MACHINE ENVIRONMENT” filed May 10, 2012,which application is herein incorporated by reference.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to the following patent applications, eachof which is hereby incorporated by reference in its entirety:

U.S. patent application titled “SYSTEM AND METHOD FOR PROVIDING ATRANSACTIONAL COMMAND LINE INTERFACE (CLI) IN A NETWORK NETWORK”,application Ser. No. ______, filed ______, 2013 (Attorney Docket No.ORACL-05377US1);

U.S. patent application titled “SYSTEM AND METHOD FOR SUPPORTINGCONFIGURATION DAEMON (CD) IN A NETWORK ENVIRONMENT”, application Ser.No. ______, filed ______, 2013 (Attorney Docket No. ORACL-05377U52);

U.S. patent application titled “SYSTEM AND METHOD FOR SUPPORTINGPERSISTENT SECURE MANAGEMENT KEY (M_KEY) IN A NETWORK ENVIRONMENT”,application Ser. No. ______, filed ______, 2013 (Attorney Docket No.ORACL-05377U54);

U.S. patent application titled “SYSTEM AND METHOD FOR SUPPORTING SUBNETMANAGER (SM) MASTER NEGOTIATION IN A NETWORK ENVIRONMENT”, applicationSer. No. ______, filed ______, 2013 (Attorney Docket No.ORACL-05377US5); and

U.S. patent application titled “SYSTEM AND METHOD FOR SUPPORTING ADRY-RUN MODE IN A NETWORK ENVIRONMENT”, application Ser. No. ______,filed ______, 2013 (Attorney Docket No. ORACL-05377US6).

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

1. Field of Invention

The present invention is generally related to computer systems andsoftware, and is particularly related to supporting a networkenvironment.

2. Background

The interconnection network plays a beneficial role in the nextgeneration of super computers, clusters, and data centers. Highperformance network technology, such as the InfiniBand (IB) technology,is replacing proprietary or low-performance solutions in the highperformance computing domain, where high bandwidth and low latency arethe key requirements.

Due to its low latency, high bandwidth, and efficient utilization ofhost-side processing resources, IB technology has been gainingacceptance within the High Performance Computing (HPC) community as asolution to build large and scalable computer clusters.

SUMMARY

Described herein are systems and methods for supporting subnetmanagement in a network environment. The network environment can includea plurality of configuration daemons (CDs), wherein each CD resides on aswitch in the network environment. The CD operates to receive aconfiguration file that includes a list of known management key (M_Key)values. Furthermore, the CD operates to store the configuration file,and make the configuration file available to a local subnet manager(SM)on the switch, wherein the local SM is associated with a currently usedM_Key value. Then, the CD operates to update the local SM with a newM_Key, after receiving an instruction from a master CD that isassociated with a master SM in the network environment.

BRIEF DESCRIPTION OF THE FIGURES:

FIG. 1 shows an illustration for supporting a management key (M_Key)protection model in an IB network, in accordance with an embodiment ofthe invention.

FIG. 2 shows an illustration for supporting a transactional command lineinterface (CLI) in a network environment, in accordance with anembodiment of the invention.

FIG. 3 illustrates an exemplary flow chart for supporting atransactional command line interface (CLI) in a network environment, inaccordance with an embodiment of the invention.

FIG. 4 shows an illustration for supporting subnet management usingconfiguration daemons (CDs) in a network environment, in accordance withan embodiment of the invention.

FIG. 5 illustrates an exemplary flow chart for supporting subnetmanagement using configuration daemons (CDs) in a network environment,in accordance with an embodiment of the invention.

FIG. 6 shows an illustration for supporting a management key (M_Key)configuration file in a network environment, in accordance with anembodiment of the invention.

FIG. 7 shows an illustration for supporting state synchronizationbetween a configuration daemon (CD) and a subnet manager (SM) on aswitch in a network environment, in accordance with an embodiment of theinvention.

FIG. 8 illustrates an exemplary flow chart for supporting statesynchronization between a configuration daemon (CD) and a subnet manager(SM) in a network environment, in accordance with an embodiment of theinvention.

DETAILED DESCRIPTION:

The invention is illustrated, by way of example and not by way oflimitation, in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that referencesto “an” or “one” or “some” embodiment(s) in this disclosure are notnecessarily to the same embodiment, and such references mean at leastone.

The description of the invention as following uses the Infiniband (IB)network as an example for a high performance network. It will beapparent to those skilled in the art that other types of highperformance networks can be used without limitation.

Described herein are systems and methods that can support subnetmanagement in a network, such as an IB network.

InfiniBand (IB) Network and Management Key (M_Key) protection model

The IB networks can be referred to as subnets. An IB subnet can includea set of hosts interconnected using switches and point-to-point links.Additionally, the IB subnet can include at least one subnet manager(SM), which is responsible for initializing and bringing up the network,including the configuration of all the switches, routers and hostchannel adaptors (HCAs) in the subnet.

FIG. 1 shows an illustration of supporting a management key (M_Key)protection model in an IB network, in accordance with an embodiment ofthe invention. As shown in FIG. 1, a management key, such as an M_Key102, can be used to protect an IB fabric, such as an IB subnet 100. Thevalues for the M_Key 102 may only be known by fabric administrators 110,which can have administrator access to the switches A-B 103-104 and thedesignated subnet manager (SM) nodes 101.

In the IB subnet 100, a secure HCA firmware implementation in HCA121-124 can keep the type and identity of various fabric nodes welldefined. Each of the HCA121-124 can implement a subnet management agent(SMA) component, e.g. SMAs 131-134. Each of the SMAs 131-134 can beassociated with an M_Key, e.g. M_Keys 141-144.

Furthermore, the connected switches A-B 103-104 can be controlled by thefabric administrator 110, which can install new M_Key values 102,out-of-band, on switches 103-104. Thus, any rogue SMA implementation131-134 may not compromise the fabric administrator 110 defined M_Key102 values that are used in the IB subnet/fabric 100.

Additionally, the fabric administrator 110 can ensure that there isinfinite M_Key 102 lease time on the switches A-B 103-104. Thus, thehost based software 161-164, e.g. a host based subnet manager ondifferent hosts 111-114 (including an operating system 151-154), may nothijack the control of any switch A-B 103-104 in the IB subnet/fabric100.

In accordance with an embodiment of the invention, a single M_Key 102value (or a single set of M_Key values) can be used for various nodes inthe in the IB subnet 100 based on the I B specification defined accessrestrictions. The correct value for a current M_Key 102 may need to bespecified before either reading or updating the M_Key 102, since thesecure HCA firmware can ensure that the “read protected” M_Key assignedto the local HCA 121-124 is not exposed to local host based software.

In accordance with an embodiment of the invention, a designated subnetmanager 101 can ignore any HCA ports with un-known M_Key value and leavethe corresponding link not initialized. The only impact of a hijackedHCA port M_Key can be that the HCA port may not be operational, and thedesignated subnet manager 101 can prevent host based software fromcommunicating via this HCA port using normal communication, i.e.non-SMP/VL15 based communication.

Furthermore, the HCA ports may be set up with finite lease time onM_Keys 102, e.g. due to a high availability concern with the subnetmanager(s) 101 that maintains the M_Key 102 lease period.

In accordance with an embodiment of the invention, the M_Key 102 can becreated and managed by fabric administrators 110 and stored in securedmemory on switches A-B 103-104 and/or HCAs121-124. A microprocessor on aswitch A-B 103-104 or a HCA121-124 can access the memory for reading outthe M_Key 102 or writing the M_Key 102 into the memory.

Transactional Command Line Interface (CLI)

In accordance with an embodiment of the invention, a transactionalcommand line interface (CLI) can be operating in a network environment,e.g. from a network switch with the current master subnet manager (SM).

FIG. 2 shows an illustration of supporting a transactional command lineinterface (CLI) in a network environment, in accordance with anembodiment of the invention. As shown in FIG. 2, a switch 201 in anetwork environment, e.g. an IB subnet 200, can include a subnet manager(SM) 202 and a configuration daemon (CD) 203.

Furthermore, the switch 201 can provide a transactional CLI 204, whichallows a user to configure and manage the IB subnet 200 using one ormore transactions based on M_Key 205. As shown in FIG. 2, a transaction210, which allows the system to perform various operations, can groupone or more commands 212 between a start command 111 and a commitcommand 213.

In accordance with an embodiment of the invention, a single upgradetransaction, e.g. the transaction 210, may not perform conflictingoperations. This can be implemented in a dynamic way where the initialoperation(s) within the transaction 210 can define both the type oftransaction 210 and which operations are allowed. For example, the startcommand 211 to initiate the transaction 210 can include parameter(s)that explicitly defines which operations are allowed.

Alternatively, a logical commit operation in a single upgradetransaction 210 can be implemented as several consecutive commitoperations, each of which can represent a sub-transaction that groupstogether a set of similar operations. For example, all remove operationscan be implemented as a sub-transaction before all add operations can beimplemented as a subsequent sub-transaction. Such an automated schemecan have the advantage of making the normal update scenarios simpler forthe user, and may also have the disadvantage of making the handling of apartially failed transaction more complex in terms of defining whichoperations have been completed and which have not.

Furthermore, the system allows the user to keep track of operations thatcan be grouped within a single transaction, in order to simplifying thehandling of a partially failed transaction in terms of defining whichoperations may have been completed and which may have not.

In accordance with an embodiment of the invention, the transaction startcommand 211 can ensure that conditions for completing and committing thetransaction are fulfilled. If the current conditions are not acceptable,the system allows the user to choose to abort the transaction.Furthermore, the user can re-issue a commit operation in force mode,after the failure of a normal commit due to current conditions.

Additionally, the command interface can generate a named template 220,which is a configuration that is not committed to the fabric, butinstead stored in a local or remote file system. This named template 220can also be generated from an already committed configuration.

The command interface allows a named template 220 to be loaded followinga transaction start command 211. The named template 220 can also be aparameter for the start command 211. The user is allowed to modify theconfiguration information from the loaded named template 220 beforecommitting it in the same way as any other configuration update.

Furthermore, the system can prepare different named templates 220 fordifferent use cases. Also, based on the named templates 220, the systemcan perform backup operations to provide protection for the system,after a complete fabric outage, which may cause the loss ofconfiguration information from all peer CD instances 206 in the fabric.Furthermore, the named template 120 allows an administrator to roll backto an earlier consistent configuration, when the administrator hascommitted an incorrect configuration. Thus, the administrator can avoidperforming a potentially very complex sequence of explicit “undo” and“redo” operations.

In accordance with an embodiment of the invention, by not allowing theoperation of changing current M_Key value 205 and the operation ofdeleting the (old) current M_Key value 205 in the same transaction, thesystem can avoid a race condition when a transaction fails into a statewhere one SM may have a current M_Key that is unknown to one or more ofthe other SM instances.

Appendix A provides further information with respect to providing atransactional command line interface (CLI) in a network environment andvarious aspects of the platform described throughout this disclosure.The information in Appendix A is provided for illustrational purposesand should not be construed to limit all of the embodiments of theinvention.

FIG. 3 illustrates an exemplary flow chart for supporting atransactional command line interface (CLI) in a network environment, inaccordance with an embodiment of the invention. As shown in FIG. 3, atstep 301, the system can provide a transactional interface at a networkswitch, wherein the transactional interface allows a user to interactwith the network environment using a transaction. Then, at step 302, thesystem can group one or more operations in the transaction. Furthermore,at step 303, the system can ensure that no conflicting operations can beincluded in the transaction.

Configuration Daemon (CD)

FIG. 4 shows an illustration of supporting subnet management usingconfiguration daemons (CDs) in a network environment, in accordance withan embodiment of the invention. As shown in FIG. 4, a networkenvironment, e.g. an IB subnet 400, can include a plurality of networkswitches, e.g. switches A-E 401-405.

Each of the network switches A-E 401-405 can include a subnet manager(SM) and a configuration daemon (CD). For example, switch A401 caninclude a CDA421 in addition to a SM A 411. Here, SM A 411 is the masterSM that is responsible for managing the subnet 400. Additionally, eachof the switches B-E 402-405 includes a standby SM, e.g. one of SMs B-E412-415, and a CD instance, e.g. one of CDs B-E 422-425.

Before the system allows an operation to be performed on the IB subnet400, the master CD A 411 can perform a consistency check for variousstate information on the peer CD instances B-E 422-425. Such stateinformation can include availability, compatible implementation/protocolversion, correct current daemon run-time state, and correct currentconfiguration state/version.

Then, the master CD A 411 can replicate a new management key (M_Key)configuration file 410 to the peer CDs B-E 422-425, before instructingthe CDs B-E 422-425 to update the related SMs B-E 412-415 with a currentM_Key value.

In accordance with an embodiment of the invention, the system can ensurethat the master SM 411 always have a known M_Key list 430 that is longerthan, or at least as long as, any other list maintained by a standby SMinstance. Also, the order of replication can ensure that a new shorterlist is applied to the standby SMs before it is applied to the masterSM.

Thus, the master SM may never have a known M_Key list 430 that isshorter than any of the standby SM instances, and the system can use thelength of the known M_Key list 430 as a criteria for determiningmastership.

Furthermore, the order of replication can be based on the type of thetransaction. When the transaction is an “addonly” transaction, the newconfiguration file can be used to update the known lists of the local(master) SM first, followed by the standby SM instances. If thetransaction is not “addonly”, then the local (master) SM is updated withthe new known lists after the new known lists have been replicated tothe standby SMs.

The replication process is considered successfully completed, after allpeer CDs have confirmed receiving and storing the configuration file,and the configuration file have passed the checksum and metadata checks.Then, the master CD A421 can complete the commit operation byinstructing each peer CD B-E 422-425 to update the related SM B-E412-415 with the new current M_Key value.

In accordance with an embodiment of the invention, the master CD A421can update the local SMA 411 after all peer CDs B-E 422-425 have beensuccessfully updated. This sequence can be independent of the particulartype of a transaction.

Additionally, the system may not instruct the SM instances to change thecurrent M_Key value until all the SM instances, i.e. the actual SMinstances—not just the corresponding CD instances—have received theupdated known M_Key list. Thus, a new current M_Key value can always beknown to all SMs, even when there is a transaction failure whileupdating the current value.

In accordance with an embodiment of the invention, an active CD, e.g. apartition daemon (PD) on an Oracle NM2 network switch, may not acceptany commit or start command when the local SM node configuration is notaugmented with hostname and internet protocol over InfiniBand (IPOIB)address list. Also, the CDs in the SM node list may need to have anappropriate firmware version that allows a start and/or commitoperation.

Furthermore, the switches in the system (i.e. including the switches inthe SM node list) need to have an appropriate firmware version beforeenabling the use of a secure M_Key setting. Configuring secure M_Key ina system with an incompatible firmware version, e.g. an older NM2firmware, can cause some local operations, such as enabling/disablinglocal ports, to become dysfunctional. These dysfunctional localoperations can include both the enable/disable operations based onexplicit CLI commands, and automated enable/disable operations that areperformed during system boot and cable hot-plug.

In accordance with an embodiment of the invention, the system can logand report the progress of various operations on the SM nodes. If thecommit operation is not started or has not fully completed for all SMnodes, then the relevant information can be logged and displayed as awarning. Also, when the system uses a “force” option, the operation maybe reported as either “fully successful” or “partially successful”. Onthe other hand, when the system does not use the “force” option, theoperation may not be reported as “partially successful”. Furthermore, ifan SM node is lost during a commit operation (i.e. it was present whenthe commit was commenced) then the operation may report a failure, andthe user may then have the option to perform another commit operation,or perform a force commit operation, with the same workingconfiguration.

Additionally, the system can perform force operations, even when some SMnode defined CD instances are not available.

For example, when an SM node instance was missing at one point in timeand then comes back as an incompatible SM node instance, e.g. an olderversion NM2 node. This system may require the problematic instance to beremoved from the SM node list, or be upgraded back to a compatible SMnode instance, before any subsequent secret M_Key configuration changecan be made.

In order for a stateful fail-over to work, e.g. with a SM enabled olderversion NM2, there may not be any use of secret M_Keys in the fabric.The system can perform a disablesecretmkey CLI command before thedowngrade, or perform a disablesecretmkey CLI command with “force”option after the downgrade, as a possible workaround for this case oftemporarily downgrading SM node instance.

Additionally, the disablesecretmkey CLI command with “force” option canalso be a way to recover a SM node when downgrade already has takenplace.

FIG. 5 illustrates an exemplary flow chart for supporting subnetmanagement using configuration daemons (CDs) in a network environment,in accordance with an embodiment of the invention. As shown in FIG. 5,at step 501, the system can provide a plurality of configuration daemons(CDs) in the network environment, wherein a master CD is an active CD ona switch with a master subnet manager(SM). Then, at step 502, the masterCD can perform consistency check on one or more states associated withone or more peer CDs in the network environment, wherein each said peerCD is an active CD on a switch with a standby SM. Furthermore, at step503, the master CD can replicate a configuration file to the one or morepeer CDs, wherein the configuration file includes at least onemanagement key (M_Key) value.

The Management Key (M_Key) Configuration File

In accordance with an embodiment of the invention, a management key(M_Key) configuration file 610 can be transmitted between theconfiguration daemons (CDs). The management key (M_Key) configurationfile can include a trusted management key (M_Key) value that ispresented to a user, and one or more key values that are transparentlygenerated based on the trusted M_Key. Additionally, the M_Keyconfiguration file can include one or more of a fixed function, functionparameter, and a locally generated random number.

FIG. 6 shows an illustration of supporting a management key (M_Key)configuration file in a network environment, in accordance with anembodiment of the invention. As shown in FIG. 6, a switch 601 in thenetwork environment 600 can include a subnet manager (SM) 602 and aconfiguration daemon (CD) 603. Furthermore, the switch 601 can provide acommand line interface (CLI) 604, e.g. a “smsubnetprotection” CLIinterface, to the user.

While the CLI interface 604 may only present a single set of managementkey (M_Key) values, e.g. a trusted M_Key 605 value, to the user, theunderlying implementation in the switch 601 can use additional sets ofkeys, which can be included in an M_Key configuration file, e.g. the “PD_(—) M _Key-config.conf” file for NM2 network Switch.

For example, the M_Key configuration file 610 can include: the trustedM_Keys 611 used for nodes defined as “trusted”, the untrusted M_Keys 612used for nodes that are considered not trusted, and the subnet managerkeys (SM_Keys) 613 used for basic SM-SM authentication and negotiation.

The handling of different M_Keys for trusted and un-trusted nodes canensure that trusted M_Keys 611 used for trusted nodes (switches inparticular) can not be observed/learned by un-trusted nodes. Thismechanism can also apply to the cases where the trusted nodes areexplicitly authenticated via encryption based protocols whereas theun-trusted nodes are not authenticated at all.

The M_Key configuration file 610 can contain a list of symbolic and hexM_Key values. Also, the M_Key configuration file 610 can contain anexplicit identification of the current M_Key value. In addition to theM_Key information, the M_Key configuration file 610 can also includemetadata information including format revision, time stamp and ID (e.g.NM2 hostname+IP) for last update, update revision number and checksum.

Furthermore, the M_Key configuration file 610 can record each addedvalue as a list of associated values. Also, the CD instance 603 can keeptrack of the current status for the sets of M_Key values, such asinformation on all the M_Key values with a specific type. For example,the M_Key configuration file 610 can include multiple lines with threevalues in each line, where each position in the line defines a key typeand a complete line can be defined as “current” or “new-current”.

In accordance with an embodiment of the invention, the management key(M_Key) configuration file 610 can be encrypted using a password 620(e.g. with a default value). The system can use the encrypted transfer,which is based on a common private password on each node, in order toensure that the sensitive data is not available to anyone being able toobserve data traffic on a management network connecting different CDinstances.

Furthermore, when an “add” command is executed, the trusted M_Key value605 is what a user of the CLI 604 relates to. Then, the system cangenerate the other key values transparently for the CLI user. Theadditional values can be generated as the sum of the user specified keyvalue and a 64 bit value generated based on a unique permutation of thecurrent password. For example, the at least eight characters in thecurrent password can be permutated in a defined way for each keytype—e.g. rotate one, rotate two.

In accordance with an embodiment of the invention, the system can ensurethat any formulas or parameters used to generate additional key valuesare not part of the information to be replicated and synchronized amongthe SM nodes. For example, the M_Key values can be generated as part ofpreparing the update transaction information, without relying on anystate for generating keys to be present among the CD or the SM instanceswhen the configuration has been defined.

Furthermore, new M_Key values can be generated as a function of both acurrently defined encryption password and a random value in order tominimize the risk of someone deducting an original password or anoriginal M_Key if either password or original M_Key should becomeavailable to someone already knowing the derived M_key value.

For example, in addition to the sum of the two 64 bit values, a platformgenerated random number can be included in the sum, in order to preventgiving away the password value. Thus, a party who knows both anun-trusted M_Key and a trusted M_Key encrypted with the password willnot be able to generate the corresponding password or trusted M_Key viaa very limited number of arithmetic operations.

Additionally, the use of incorrect password, e.g. as a result of notupdating the password correctly on all SM nodes, can cause a checksumerror on the remote side. In order to differentiate the type of checksumerror due to incorrect password from the type of checksum error due tofile corruption during transfer, there can be a checksum for both theencrypted file and a checksum for the original file version. Here, achecksum error for the encrypted file can indicate a transfer error,whereas a checksum error for the un-encrypted file can indicate anincorrect password. Also, there can be metadata for both the encryptedand un-encrypted files in order to ensure consistency and multipleversion compatibility at both levels.

In accordance with an embodiment of the invention, prior to initiating atransaction, the system can ensure consistent password encryption foreach CD.

A challenge for verifying the password encryption is that the passworditself may not be communicated explicitly across the potentiallyinsecure management network. Instead, the system can use the password toencrypt a well known message string (i.e. well known “a priori” by allSM nodes). Then, the verification of consistency of the current passwordcan be achieved by sending the encrypted version of the well knownmessage and having the peer SM nodes to confirm that the decryption ofthe message results in the original well known message.

For example, the well known message can be the current password stringitself. Furthermore, a dedicated well known string can be a more robustscheme, since the password string typically has a limited length.Additionally, by maintaining a list of historical password or keyvalues, it is possible for the verification logic to check with olderpassword/key values if the current one fails and thereby identify a caseof incomplete updates among the SM nodes. Also, a remote check failurefor the current value could lead to encryption and checking of oldervalues to determine whether the remote SM node has yet been updated withthe current password value.

In accordance with an embodiment of the invention, such a verificationmethod can also be used to verify that current public/private keyinfrastructure is consistent among the involved SM nodes, i.e. to ensurethat no accidental update has taken place followingauthentication/verification during discovery/re-discovery. Ifpublic/private key infrastructure is in place, then this can be used forencryption of replicated configuration information in order to avoidmaintaining a separate encryption password on each SM node.

Start Logic and Consistency Check

In accordance with an embodiment of the invention, when a transactionstart command is executed from a current master node, the stateinformation on various subnet manager (SM) nodes can be verified so thatany inconsistencies in the state information can be detected. Also, apre-condition for an update transaction can be that the system has aconsistent (i.e. commit completed and correct checksums) configurationfile with the same configuration update revision number.

The administrator can ensure that correct configuration can be carriedforward in an IB subnet, following a failed update transaction that maycause the state information on the SM nodes to be inconsistent. Theadministrator can initiate a new update transaction to bring the variousSM nodes in synchronization. Additionally, an administrator can verifythat the inconsistent state does not represent any unexpected condition,and can perform the diff operations that can explicitly provide thedetailed difference in various SM nodes or between a specified pair ofSM nodes.

Furthermore, when the local configuration on the current master SM nodeis not the most recent one or is incorrect, the system can perform aspecial operation from the master SM node to fetch the currentconfiguration from another standby SM node, and use this configurationas a starting point for the working configuration when performing a newupdate transaction. Additionally, a special “force” option can be usedto override consistency checks for both start and commit operations,e.g. after the inconsistent state has been analyzed and the correctconfiguration for synchronizing the various SM nodes have beendetermined by the administrator.

In accordance with an embodiment of the invention, when fatalsynchronization problems occurs in the IB subnet, the administrator canbe allowed to resolve the underlying problem before continuing orrestarting any update transaction. For example, when performing a commitoperation, the configuration state of the various SM nodes is expectedto be exactly the same as the configuration state when the correspondingstart operation was performed, i.e. any change in this period may implythat some unsolicited state change has taken place and may indicate afatal synchronization error between the various SM nodes. Additionally,at both start and commit time, it can be verified that only a singlemaster is present among the SM nodes. If otherwise, then there is afatal synchronization problem among the SM nodes.

Furthermore, the consistency of encryption password value among thevarious SM nodes can be verified as part of the transaction startoperation, and can again be verified in the initial part of the commitoperation along with the verification that no configuration change hastaken place. Additionally, the state consistency checks can be availableas command line interface (CLI) subcommands from any SM node at anypoint in time.

State Synchronization between the CD and the SM

FIG. 7 shows an illustration for supporting state synchronizationbetween a configuration daemon (CD) and a subnet manager (SM) on aswitch in a network environment, in accordance with an embodiment of theinvention. As shown in FIG. 7, a switch 701 in an IB subnet 700 caninclude a subnet manager (SM) 702 and a configuration daemon (CD) 703.

When the CD 703 receives a new M_Key configuration file 710 as part of acommit transaction, the CD 703 can store the new M_Key configurationfile 710 and make the list of known M_Key values 704 available to thelocal SM 702. Then, the SM 702 can start using the new list of knownM_Key values 704 in subnet probing/discovery operations. The SM 702 maycontinue using the previously defined current M_Key value 705 whenupdating/setting M_Key value for any subnet management agent (SMA)instance.

The CD 703 may not update the current M_Key value 705 used by the SM702, prior to receiving an instruction from the master CD. Also, the CD703, when it is a standby CD, can communicate the new list of knownvalues 704 to the local SM 702, before the operation is acknowledged tothe master CD, which implements the distributed transaction.

Unlike partition configuration updates, the new list of known M_Keys 704can be made available to the SM 702 immediately, since this does notimpose any state change in the subnet, and the manual and automaticprocedures can make sure that a new list of known M_Key values 704 mayalways include any secret M_Key value that may be present in a switchnode in the system (i.e. including a switch node with a disabled SM).

Furthermore, in order for the list of known M_Key values 704 to be usedas part of the consistency check and master election among secret keyenabled SMs in the subnet, the order of updates can ensure that thecurrent master always has the largest list of known M_Key values, andthereby avoid any un-intended master handover in the middle of an updatetransaction.

In accordance with an embodiment of the invention, the CD instance 703can update the local state information about the current M_Key value705, when the CD instance 703 is instructed by the master CD, whichcontrols the commit operation. Also, the CD instance 703 can instructthe local SM 702 to start using the new current M_Key value, e.g. via adedicated SM CLI command.

Furthermore, whenever the SM 702 or CD 703 is restarted, the CD instance703 can update the SM 702 with the current local secret keyconfiguration. Thus, the system can ensure that the subnet managementoperation is always consistent with the current configuration, and thesystem can ensure that no master election is performed without thecurrent secret key configuration in place. Also, the statesynchronization between CD 703 and SM 702 during startup can make surethat the SM 702 has the current configuration prior to initiating theinitial subnet discovery.

In accordance with an embodiment of the invention, the CD 703 can use adedicated SM CLI command to communicate the new list of known M_Keyvalues 704 to the SM 702. For example, the SM 702 may only receive thesecret key configuration via the private CLI commands, and may notperform any persistent storing of this configuration (i.e. the secretkey configuration may only be part of the SM run-time image).

The SM CLI command can include a plain list of comma separated hex M_Keyvalues that defines the list of known M_Key values 704. Furthermore, theSM CLI allows three separate lists to be specified, in order to takethree independent M_Key values as input, such as the trusted M_Key, theuntrusted M_Key and the SM_Key.

For example, in a NM2 switch, the new current M_Key value can be thesame as what was identified as “new current” in the replicated M_Keyconfiguration file. The old current M_Key value can remain to be part ofthe known M_Key list. (I.e. two update transactions may be needed inorder to delete a value that has been used as current.)

In order to ensure that a new current M_Key value is not activatedprematurely, the replicated configuration file may always containinformation about the (still) currently used M_Key value as well as theintended new value. When a set-current operation takes place (i.e. thecorresponding command is received from the master CD implementing thetransaction), the new M_Key value can be signaled to the SM 702, andthen the recorded currently used M_Key value can be updated to reflectthe new specified value before the operation is acknowledged. Thus, themaster SM can have a known M_Key list longer than, or at least as longas, any other list maintained by a standby SM.

Furthermore, the CD instance 703 can keep track of both the actualcurrent M_Key value 705 and the enabled state. For example, the handlingof the disabling and enabling of secret M_Key for the SM 702 can bereflected via a value of zero for the disabled state, and a non-zerocurrent value for the enabled state. Furthermore, the CD instance 703can provide the value of zero, instead of the current value, to the SM702, when the system is set to be in a disabled state. Then, the CDinstance 703 can signal the (real) current M_Key value to the SM 702,when the system is changed from a disabled state into an enabled state.

In accordance with an embodiment of the invention, the SM 702 canreceive explicit enable/disable notifications for the use of secretM_Keys. Also, the SM can use a readable M_Key as defined in aconfiguration file, e.g. the opensm.conf file, when the use of secretM_Key is disabled (as a default). Additionally, the readable M_Key valuein the configuration file can be maintained by the CLI commands.Furthermore, the M_Key protection bits to be used by the SM 702, whichreflects readable M_Key, can be defined in the opensm.conf configurationfile. On the other hand, the M_Key protection bits can be implicitlydefined when the “secret” mode is enabled via the SM CLI interface.

Also, the M_Key lease time value can be a constant defined in theconfiguration file. For example, a constant defined by the NM2 FWrevision specific opensm.conf template can be sufficient. Alternatively,the M_Key lease time value can be configurable. This M_Key lease timevalue can be large (e.g. in the order of at least one minute) so thatthere is never any practical risk that the SM can not refresh the leasetime before the deadline (i.e. as long as at least one SM is sweeping).Additionally, the platform level monitoring daemons can also have a rolein refreshing the local lease time.

FIG. 8 illustrates an exemplary flow chart for supporting statesynchronization between a configuration daemon (CD) and a subnet manager(SM) in a network environment, in accordance with an embodiment of theinvention. As shown in FIG. 8, at step 801, a configuration daemon (CD)on a switch in the network environment can receive a configuration filethat includes a list of known management key (M_Key) values.Furthermore, at step 802, the CD can store the configuration file, andmake the configuration file available to a local subnet manager(SM) onthe switch, wherein the local SM is associated with an currently usedM_Key value. Then, at step 803, the CD can update the local SM with anew M_Key, after receiving an instruction from a master CD that isassociated with a master SM in the network environment

Failed Update Transaction and Master Subnet Manager (SM) Failover

When a update transaction is successfully completed, the configurationdaemons (CDs) on all subnet manager (SM) nodes can have the sameconfiguration, e.g. with the same configuration update revision number.Additionally, the CDs can either have the same current M_Key value orhave no new current M_Key value.

On the other hand, a failed update transaction may cause an availabilityor configuration issue with the current SM nodes in an IB subnet. Forexample, when an update transaction fails during the replication phase,the CDs may have different current update revision number values (andwith different file checksums). Also, if the replication phase iscompleted successfully, but the update of current M_Key value is notcompleted, then the CDs may have the same update revision number valuesand same basic file checksums, but with the different current M_Keyvalue.

In these scenarios, the administrator can use a CLI command provided bythe SM Node switches, e.g. the listactive command provided by the NM2switches, in order to discover and resolve the current status. Then, theadministrator can decide how to perform another update transaction onthe current master CD, in order to bring the system to a well definedstate, e.g. using force options and updating the list of SM nodes in thesystem.

If a failed transaction leaves the current persistent configuration withboth an old current M_Key value and a new current M_Key value, then anynew transaction can start out in this state, and behave as if aset-current CLI command had been executed for the M_Key value that isreflected as “new” in the current configuration file. Furthermore, theuser can be aware of this current status, and is allowed to modify thesetting if needed. Additionally, a failure to observe the rules for whatoperations can be performed following a failed transaction can lead toan incorrect master election in the subnet, e.g. causing either multipleor no master SMs in the subnet.

In accordance with an embodiment of the invention, the transaction startcommand can present all relevant state information for the configurationon all SM nodes to the user. Also, the transaction start command canmake sure that the user is aware of any inconsistencies, e.g. after afailure of an updated transaction. Furthermore, the transaction startcommand can perform special force actions in order to initiate atransaction, and allows the user to select the starting pointconfiguration from any of the available SM nodes.

If the active master is operational after a failure of an updatedtransaction, the first step to handle the failed update transaction isto check availability and configuration of other SM nodes, potentiallybased on error logs/messages from the failed update transaction. If theproblem can be corrected, then the same transaction can be retried (i.e.without use of force mode). If the problem can not be corrected (e.g. anSM node—potentially the old master—is no longer operational/available),then any delete operation may not be re-tried in force mode, since thisoperation may delete a key value that may still be in use by an enabledSM.

In accordance with an embodiment of the invention, a disable operationcan be retried in force mode, since the disable operation does notchange the current known key list and does not impose any danger ofincorrect master election procedures. Furthermore, depending upon the SMpriorities and globally unique identifiers (GUIDs) of the currentlyavailable SM nodes relative to the failed SM node(s), the disabled orenabled state of the system may change as the failed SM nodes becomeoperational again.

Furthermore, force retrying a failed enable operation is safe, since itdoes not impose any danger of incorrect master election procedures.Also, the state of the system in terms of enabled/disabled state of themaster may not be certain when the same key configuration is present.

Additionally, force retrying an add operation is safe, since it mayincrease the number of known keys and thereby it can ensure that the SMnodes have received this update and may take precedence in terms ofmaster election relative to the currently unavailable SM node(s).

In accordance with an embodiment of the invention, by setting a newlyadded key as current, the system can be configured for fencing off a SMon the unavailable SM node(s). Thus, the SM on the unavailable SMnode(s) may not influence the state of any node (potentially includingitself) that has received the updated secret M_Key value.

The system can use force-add and set-current operations to ensure thefencing off an SM node with un-defined state, without any need forphysical service actions. Such fencing mechanism allows the SM nodesconfiguration to be changed, and also allows the current partitionpolicy to be updated.

When a leaf switch is in an undefined state that can cause thepartitioning state of the directly connected hosts to be notcontrollable, the partitioning configuration may not be updated orchanged in a way that conflicts with the current configuration that isassumed to be implemented by an active SM on that leaf switch.

In order to update or change the partitioning configuration, the leafswitch can be brought to a well defined state with no active SM, or theSM can be brought in synchronization with the rest of the SM nodes, viasome additional service actions.

In accordance with an embodiment of the invention, the system cansupport master SM failover in the middle of an M_Key update transaction,which follows the same ACID transaction principles as partition policyupdates.

The system can distribute the new list of known M_Key values first, anddefine a new M_Key value as current for all the SMs in the final part ofthe commit operation. Also, the enhanced master election protocol canensure that the elected master can always be the SM that has the longestlist of known key values and can be best suited to handle the currentconfiguration.

Thus, the system can ensure that there may never be a case where a newmaster SM is not be able to discover/configure the subnet because oflack of knowledge about the most recent M_Key value. For example, all SMenabled switch nodes can have the same new policy if the policy isenabled on any one of switch nodes, otherwise if the policy has not beenreplicated to all SMs, then no SM can have the new policy enabled.

The present invention may be conveniently implemented using one or moreconventional general purpose or specialized digital computer, computingdevice, machine, or microprocessor, including one or more processors,memory and/or computer readable storage media programmed according tothe teachings of the present disclosure. Appropriate software coding canreadily be prepared by skilled programmers based on the teachings of thepresent disclosure, as will be apparent to those skilled in the softwareart.

In some embodiments, the present invention includes a computer programproduct which is a storage medium or computer readable medium (media)having instructions stored thereon/in which can be used to program acomputer to perform any of the processes of the present invention. Thestorage medium can include, but is not limited to, any type of diskincluding floppy disks, optical discs, DVD, CD-ROMs, microdrive, andmagneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flashmemory devices, magnetic or optical cards, nanosystems (includingmolecular memory ICs), or any type of media or device suitable forstoring instructions and/or data.

The foregoing description of the present invention has been provided forthe purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Many modifications and variations will be apparent to the practitionerskilled in the art. The embodiments were chosen and described in orderto best explain the principles of the invention and its practicalapplication, thereby enabling others skilled in the art to understandthe invention for various embodiments and with various modificationsthat are suited to the particular use contemplated. It is intended thatthe scope of the invention be defined by the following claims and theirequivalence.

Appendix A A Sample CLI Interface

For example, the Oracle NW2 network switch can provide asmsubnetprotection CLI interface to support the transactional protocolas described in the above sections. The smsubnetprotection CLI interfacecan provide the following commands:

start

The start command can check (via CD) whether all required CD instances(“smnodes”) are available and operational. If so, the system canestablish a working copy of the current M_Key configuration. The startcommand can include options to force start, to implicitly enable, and tospecify a type (e.g. “addonly” or “deleteonly”) for the transaction.

If force option is chosen, the operation can succeed if not all CDs areavailable, but a warning to the user as well as a log entry identifyingthe set of “missing” CDs can be made if not all CDs are available.

In order for the start command to succeed, the enabled status for theuse of secret M_Key may need to be set. The enabled status can initiallybe set via using the enablesecretmkey command, or via choosing the“enable” option that can be specified for the start command. The reasonfor having the “enable” option for the start command is to ensure thatsecret M_Key usage can be enabled together with a defined current valueand/or list of known values.)

Furthermore, if “addonly” type is specified for a transaction, then nodelete operations is allowed in this transaction (vice versa if“deleteonly” is specified). It is also not legal to specify both“deleteonly” and “addonly” for the same transaction. If neither“deleteonly” nor “addonly” is specified, then only a set-currentoperation is allowed. On the other hand, the set-current operation canbe used independently of whether the transaction is “addonly” or“deleteonly”.

Listactive

The listactive command can display the last committed list of M_Keys aswell as the current M_Key value and the enabled status. This command canbe invoked independently of any on-going transaction. The listactivecommand can include options for listing symbolic and/or hex.

listworking

The listworking command can display the resulting list and current baseof changes since the last start command (i.e. that has not beencommitted). The listworking command can include options for listingsymbolic and/or hex.

Listlocalmkey

The listlocalmkey command can display the current M_Key value (if any)defined for local switch chip (e.g. an IS4 instance). The output canreflect if any M_Key is defined, and if so if it is “readable” or“secret”. Additionally, the output can reflect the case where no M_Keyis used.

In the case of configuration errors (e.g. provoked by manual access tothe sysfs interface), the output can also reflect illegal, orun-supported, protection bit values. Also, the output can reflect thatthe NM2 SM is not able to manage this node, e.g. when the M_Key value iszero, but the protection bit value indicates a “secret” mode.

This operation can be performed from any NM2 independently of local SMenabled or master status.

Setlocalsecretmkey

The setlocalsecretmkey command can set the M_Key value for the local IS4instance to the specified value and set the associated mode to “secret”(i.e. the M_Key protect bits). The setlocalsecretmkey command caninclude options for specifying symbolic with password requirements orplain hex.

This operation can be performed from any NM2 node, independently oflocal SM enabled or master status. The value zero is not a legal secretM_Key value, and the CLI command can reject it.

Add

The add command can add a new M_Key value to the working list. The addcommand can include options for specifying symbolic with passwordrequirements or plain hex. The add command is only legal if thetransaction has been started with “addonly” option.

The revoking of any incorrectly added value(s) can be accomplished byexplicitly aborting the current transaction via the abort subcommand.This ensures that the adding and then the setting current of a newlyadded value is cleaned up in a consistent and well defined way.

Delete

The delete command can remove an existing M_Key value from the list. Thedelete command can include options for specifying symbolic or plain hex.

This command is only legal if the transaction is started with“deleteonly” option. Furthermore, it is not legal to remove the“current” value, nor the “old current” value, which is enforced by theinterface. Also, it is not legal to delete any value that may stillexist as “current” for any NM2 node in the system that may betemporarily offline at the moment. This constraint may not be enforcedby the interface, but can be documented as a rule that the administratorshould follow.

Additionally, the revoking of any incorrectly deleted value(s) can beaccomplished by explicitly aborting the current transaction via theabort subcommand.

Set-current

The set-current command can set an already defined M_Key value as thecurrent M_Key value that is used by the master SM following a subsequentcommit operation. The set-current command can include options forspecifying symbolic or plain hex.

When updating current M_Key value, then the old current M_Key value canbe recorded. It is not legal to both update the current M_Key value anddelete the old current M_Key value in the same update transaction. Also,multiple set-current operations during the same transaction can be legalas long as they all identify a currently defined M_Key value, and onlythe last operation prior to commit may have any impact on the resultingactive configuration.

If no current M_Key value is defined when a configuration is committed,then this is accepted, but the result is the same as if theconfiguration had been disabled.

Once a current M_Key value is defined via the set-current command, thenit is not possible to unset it unless the configuration is disabled.This also implies that it is not possible to add more known values whilea configuration is disabled.

Thus, in order to update the known list when in disabled mode, theconfiguration has to be temporarily enabled while the configuration isupdated, but can then be immediately disabled again afterwards.

Commit

The commit command can make the configuration that has been (re)definedsince the last start operation active in the system via the PD-PD commitprotocol. The commit command can include option for force commit.

By default, the operation can fail if not all defined (“smnodes”) PDinstances are present. If “force” option is specified, then theoperation may still succeed if not all PD instances are available but awarning and log message can be generated.

Abort

The abort command can discard all operations that have not beencommitted since the last start command without any impact on the activeconfiguration. Any incorrect argument to delete or add commands can berevoked by performing an abort operation.

Setpassword

The setpassword command can define a password to be used for encryptingM_Key list before replicating to other nodes. Furthermore, same passwordcan be defined on all “smnodes” NM2s and can be updated in the samecontrolled way as when updating “smnodes” membership.

The password with an alphanumeric string of at least 8 characters isconsidered strong. The system may provide a built in-default passwordvalue, which can be replaced via the setpassword command.

The method for encryption can be based on a open source toolkit, wherethe user supplied password can be used as input to theencryption/decryption algorithms.

The specified password can also be used to generate derived key valuesbased on key values specified by the add command. The specified passwordcan be persistently recorded locally on the NM2, until it is explicitlyupdated again.

Whenever a password has been updated on one “smnode”, it has to beupdated on the other “smnodes” before any transaction can take place.

Enablesecretmkey

The enablesecretmkey command can represent an implicit start and committransaction for the currently defined configuration. Theenablesecretmkey command can include option for “force start/commit”.

If no configuration is currently defined, then the command may have noeffect, except for verifying that all “smnodes” peers are available andin the correct state, and also setting the enabled status to “enabled”.(The enabled status is a dedicated flag in the underlying configurationfile.) If no configuration is currently defined, then the enabled statusmay have no effect on the operation of the SM.

Disablesecretmkey

The disablesecretmkey command can represent an implicit start /committransaction for the currently defined configuration with the side effectthat no secret M_Key value can be used by any SM on “smnodes”. Unlike anordinary transaction, the start /commit transaction does not update thecurrent M_Key value defined by the configuration, but sets the “enabledstatus” to “disabled”. The disablesecretmkey command can include optionfor “force start/commit”.

The disabled status implies that the SM may either be using any alreadydefined readable M_Key value, or not use any M_Key value at all. Also,the list of known secret M_Key values can still be defined, and themaster SM can still be able to discover and manage any port/node thatwas not available during the initial update to “non-secret” M_Key, butthat is now available and with one of the known secret M_Key valuesactive.

The update transactions, e.g. the transactions caused by commit,enablesecretmkey or disablesecretmkey subcommands, can be implemented ina way that is safe in terms of that the master election state within anoperational subnet with NM2 2.1 firmware on all SM enabled NM2s can bewell defined also if the transaction fails.

Furthermore, a failed transaction may be repeated in order to try tocomplete it successfully in the case of a transient problem without anyinherent risk of more dramatic inconsistencies. Also, fencingoperations, such as defining an additional known key value as well asmaking it current, can be done in force mode to ensure that an NM2 nodewith unknown state may not be able to modify the state of the subnet.

Additionally, the disabling use of secret keys following a failed updatetransaction, or the deletion of known key values following a failedtransaction may lead to inconsistent master election in the subnet andmay therefore not be attempted before the set of smnodes have not beenbrought to a consistent state.

Here, bringing the system to a consistent state can involve performing aenablesecretmkey or an update transaction from the current master, basedon whether the (possibly new) master has the updated (desired) currentconfiguration.

What is claimed is:
 1. A method for supporting network management in anetwork environment, comprising: receiving, via a configuration daemon(CD) on a switch in the network environment, a configuration file,wherein the configuration file includes a list of known management key(M_Key) values; storing the configuration file, and making theconfiguration file available to a local subnet manager(SM) on theswitch, wherein the local SM is associated with an currently used M_Keyvalue; and updating the local SM with a new M_Key, after receiving aninstruction from a master CD that is associated with a master SM in thenetwork environment.
 2. The method according to claim 1, furthercomprising: including in the configuration file a trusted M_Key and oneor more key values that are transparently generated based on the trustedM_Key.
 3. The method according to claim 1, further comprising:continuing using the old current M_Key when updating/setting M_Key valuefor any subnet management agent (SMA) instance until the local SM isupdated with the new M_Key.
 4. The method according to claim 1, furthercomprising: using, via the CD, a dedicated SM command line interface(CLI) command to update the local SM.
 5. The method according to claim1, further comprising: updating, via the CD, the local SM with currentlocal secret key configuration when at least one of the SM and CD isrestarted.
 6. The method according to claim 1, further comprising:containing in the configuration file information about the currentlyused M_Key value as well as the intended new M_Key value.
 7. The methodaccording to claim 6, further comprising: recording the currently usedM_Key value in the list of known M_Key values.
 8. The method accordingto claim 1, further comprising: using the new M_Key value to fence off aSM on an unavailable node.
 9. The method according to claim 1, furthercomprising: ensuring that the master SM has a known M_Key list notshorter than any M_Key list maintained by a standby SM instance.
 10. Themethod according to claim 9, further comprising: initiating a commitoperation on the master CD.
 11. A system for supporting networkmanagement in a network environment, comprising: one or moremicroprocessors; a configuration daemon (CD) on a switch running on theone or more microprocessors, wherein the CD operates to receive aconfiguration file that includes a list of known management key (M_Key)values; store the configuration file, and make the configuration fileavailable to a local subnet manager(SM) on the switch, wherein the localSM is associated with an currently used M_Key value; and update thelocal SM with a new M_Key, after receiving an instruction from a masterCD that is associated with a master SM in the network environment. 12.The system according to claim 11, wherein: the configuration fileincludes a trusted M_Key and one or more key values that aretransparently generated based on the trusted M_Key.
 13. The systemaccording to claim 11, wherein: the local SM continues using the oldcurrent M_Key when updating/setting M_Key value for any subnetmanagement agent (SMA) instance until the local SM is updated with thenew M_Key.
 14. The system according to claim 11, wherein: the CDoperates to use a dedicated SM command line interface (CLI) command toupdate the local SM.
 15. The system according to claim 11, wherein: theCD operates to update the local SM with current local secret keyconfiguration when at least one of the SM and CD is restarted.
 16. Thesystem according to claim 11, wherein: the configuration file containsinformation about the currently used M_Key value as well as the intendednew M_Key value.
 17. The system according to claim 16, wherein: the CDoperates to record the currently used M_Key value in the list of knownM_Key values.
 18. The system according to claim 1, wherein: the newM_Key value is used to fence off a SM on an unavailable node.
 19. Thesystem according to claim 11, wherein: the master CD operates to ensurethat the master SM has a known M_Key list not shorter than any M_Keylist maintained by a standby SM instance.
 20. A non-transitory machinereadable storage medium having instructions stored thereon that whenexecuted cause a system to perform the steps comprising: receiving, viaa configuration daemon (CD) on a switch in a network environment, aconfiguration file, wherein the configuration file includes a list ofknown management key (M_Key) values; storing the configuration file, andmaking the configuration file available to a local subnet manager(SM) onthe switch, wherein the local SM is associated with an currently usedM_Key value; and updating the local SM with a new M_Key, after receivingan instruction from a master CD that is associated with a master SM inthe network environment.